The Microsoft® Windows and Macintosh® operating systems have revolutionized computer usability for many people by providing a visually-oriented computer screen that can be accessed by pointing and clicking at the objects on the screen. This graphical, window-based interface provides an intuitive replacement for the complex internal representation of the underlying computer data. The vast majority of people find the visually-based organization much easier to use, and many years of development effort have been devoted to creating an easy-to-use environment. By far, most people use a mouse device to control the computer to point and click. However, many people have a requirement for a different approach than a mouse for pointing or clicking. For example, individuals with high-level Spinal Cord Injury (SCI), ALS, or other conditions may not be able to effectively operate a device such as a mouse with their hands.
Various approaches have been presented for tracking other body features with light, such as motions of the face. For example, the IST switch from WordsPlus (see www.words-plus.com/website/pdf_files/istswtch.pdf) uses a simple infrared reflective device that can clip on to objects such as eyeglasses and detect simple gestures such as an eye blink. A slightly more complex approach, disclosed in U.S. Pat. No. 7,369,951 to Blosser and Eulenberg, similarly uses an infrared reflection, and allows for setting thresholds. With proper thresholds, this type of device can detect an eye blink or other facial movements, such as a raising a cheek or sticking out the tongue. These switches are relatively simple on/off devices. The drawback of these devices is that they tend to be too simple for some uses, and because they are general purpose (i.e., usable for many different types of motions), they are not fine-tuned to a particular type of facial gesture and thus require precise set up and calibration each time they are used. They provide only a simple on/off switch output. Since it is only an on/off approach, these devices are not very suitable for pointing input for a device such as a computer, but may be suitable for providing the click input, therefore do not replace all mouse functions. Additionally, these simple approaches are not well-suited to monitoring gestures from the mouth because they have no way of separating an intentional mouth gesture from a mouth movement naturally made during speech.
For higher precision, various approaches have been presented to help with pointing needs. For example, head trackers, such as the infrared device disclosed in U.S. Pat. No. 7,221,437 to Schaefer, have been invented to follow the motion of the head and move the computer pointer accordingly. However, for some users, moving the head is difficult or leads to discomfort. For many such users, making facial gestures or moving the eyes is an easier way to achieve the control they need.
Eye tracking devices have been invented to allow control with movements limited to the face. However, their drawbacks are in some ways the opposite of the simple optical switches described above—they tend to be complex, expensive, and difficult to use. For example, U.S. Pat. No. 4,145,122 to Rinard discloses eyeglasses containing an infrared mirror and a video camera to watch the eye. U.S. Pat. No. 4,659,197 to Weinblatt shows a device with many mechanical parts, semi-silvered mirrors, etc. U.S. Pat. No. 5,844,544 to Kahn, et. al. has lenses and a semi-silvered mirror to measure the orientation of the eye. U.S. Pat. No. 5,345,281 to Taboada discloses a device with beamsplitters and motors to track the position of the pupil. U.S. Pat. No. 5,270,748 to Katz depicts a curved surface with an array of detectors to locate a reflection from a light source and infer the eye's position from that. U.S. Pat. No. 4,720,189 to Heynen discloses the use of a spatial filter to create a rectangular pattern from the reflection from the eye. U.S. Pat. No. 5,231,674 to Cleveland, et al. and U.S. Pat. No. 4,836,670 to Hutchinson disclose image-based approaches, requiring video imagery of the eye.
A somewhat simpler approach than the above is described by Udden in U.S. Pat. No. 5,180,907, in which an internal light source is added to the light originating with another LED and reflected from the eye, so as to keep the total light level constant. This mechanism is provided to reduce the problem of calibrating the non-linear response from a simple LED/detector pair. However, it is doubtful that this somewhat simpler approach provides sufficient accuracy for use in pointing, as there are many types of nonlinearity in the disclosed system than are caused by the light response of the detectors.
The main technical difficulty that must be overcome for success with eye tracking devices in the art is that great precision is required to be able to move a pointer to the exact place the eye is looking. If there is any error, the pointer will not be exactly where the user is looking, so it will be very difficult for the user to see the pointer, resulting in the user chasing the pointer around in an attempt to look at it. With a head tracker, this is not such an issue, because if there is some error, the user simply can aim the eyes at the pointer, even if the head as a whole is not exactly aligned with the eyes. Thus, the reason the eye trackers in the prior art tend to be complex and expensive is because they are required to have very high precision to be useful. However, once working, these devices are appropriate for providing the pointing input to a device such as a computer.
To summarize, several optical technologies are available for sensing the position of parts of the face, such as the eyes. On one extreme are eye trackers, which require extreme precision and complexity to provide accurate pointing. On the other extreme are simple switch-type devices that provide only simple on/off switching input.